Logic Tensor Networks

逻辑张量网络

Samy Badreddine ${}^{\mathrm{a},\mathrm{b},\ast }$ ,Artur ${\mathrm{d}}^{\prime }$ Avila Garce ${\mathrm{z}}^{\mathrm{c}}$ ,Luciano Serafini ${}^{\mathrm{d}}$ ,Michael Spranger ${}^{\mathrm{a},\mathrm{b}}$

${}^a$ Sony Computer Science Laboratories Inc,3-14-13 Higashigotanda,141-0022,Tokyo,Japan

${}^b$ Sony AI Inc,1-7-1 Konan,108-0075,Tokyo,Japan

${}^c$ City,University of London,Northampton Square,EC1V 0HB,London,United Kingdom

${}^d$ Fondazione Bruno Kessler,Via Sommarive 18,38123,Trento,Italy

Attempts at combining logic and neural networks into neurosymbolic approaches have been on the increase in recent years. In a neurosymbolic system, symbolic knowledge assists deep learning, which typically uses a sub-symbolic distributed representation, to learn and reason at a higher level of abstraction. We present Logic Tensor Networks (LTN), a neurosymbolic framework that supports querying, learning and reasoning with both rich data and abstract knowledge about the world. LTN introduces a fully differentiable logical language, called Real Logic, whereby the elements of a first-order logic signature are grounded onto data using neural computational graphs and first-order fuzzy logic semantics. We show that LTN provides a uniform language to represent and compute efficiently many of the most important AI tasks such as multi-label classification, relational learning, data clustering, semi-supervised learning, regression, embedding learning and query answering. We implement and illustrate each of the above tasks with several simple explanatory examples using TensorFlow 2. The results indicate that LTN can be a general and powerful framework for neurosymbolic AI.

Keywords: Neurosymbolic AI, Deep Learning and Reasoning, Many-valued Logics.

Artificial Intelligence (AI) agents are required to learn from their surroundings and reason about what has been learned to make decisions, act in the world, or react to various stimuli. The latest Machine Learning (ML) has adopted mostly a pure sub-symbolic learning approach. Using distributed representations of entities, the latest ML performs quick decision-making without building a comprehensible model of the world. While achieving impressive results in computer vision, natural language, game playing, and multimodal learning, such approaches are known to be data inefficient and to struggle at out-of-distribution generalization. Although the use of appropriate inductive biases can alleviate such shortcomings, in general, sub-symbolic models lack comprehensibility. By contrast, symbolic AI is based on rich, high-level representations of the world that use human-readable symbols. By rich knowledge, we refer to logical representations which are more expressive than propositional logic or propositional probabilistic approaches, and which can express knowledge using full first-order logic, including universal and existential quantification $\left(\mathrm{\forall }x\text{and}\mathrm{\exists }y\right)$ ,arbitrary $n$ -ary relations over variables,e.g. $R\left(x,y,z,\dots \right)$ ,and function symbols,e.g. father $\mathrm{Of}\left(x\right),x+y$ ,etc. Symbolic AI has achieved success at theorem proving,logical inference,and verification. However, it also has shortcomings when dealing with incomplete knowledge. It can be inefficient with large amounts of inaccurate data and lack robustness to outliers. Purely symbolic decision algorithms usually have high computational complexity making them impractical for the real world. It is now clear that the predominant approach to ML, where learning is based on recognizing the latent structures hidden in the data, is insufficient and may benefit from symbolic AI [17]. In this context, neurosymbolic AI, which stems from neural networks and symbolic AI, attempts to combine the strength of both paradigms (see [16, 40, 54] for recent surveys). That is to say, combine reasoning with complex representations of knowledge (knowledge-bases, semantic networks, ontologies, trees, and graphs) with learning from complex data (images, time series, sensorimotor data, natural language). Consequently, a main challenge for neurosymbolic AI is the grounding of symbols, including constants, functional and relational symbols, into real data, which is akin to the longstanding symbol grounding problem [30].

*Corresponding author

Email addresses: badreddine. samy@gmail.com (Samy Badreddine), a. garcez@city.ac.uk (Artur d'Avila Garcez),

serafini@fbk.eu (Luciano Serafini), michael . spranger@sony . com (Michael Spranger)

Logic Tensor Networks (LTN) are a neurosymbolic framework and computational model that supports learning and reasoning about data with rich knowledge. In LTN, one can represent and effectively compute the most important tasks of deep learning with a fully differentiable first-order logic language, called Real Logic, which adopts infinitely many truth-values in the interval $\left[0,1\right]\left[22,25\right]$ . In particular,LTN supports the specification and computation of the following AI tasks uniformly using the same language: data clustering, classification, relational learning, query answering, semi-supervised learning, regression, and embedding learning.

LTN and Real Logic were first introduced in [62]. Since then, LTN has been applied to different AI tasks involving perception, learning, and reasoning about relational knowledge. In [18, 19], LTN was applied to semantic image interpretation whereby relational knowledge about objects was injected into deep networks for object relationship detection. In [6], LTN was evaluated on its capacity to perform reasoning about ontological knowledge. Furthermore, [7] shows how LTN can be used to learn an embedding of concepts into a latent real space by taking into consideration ontological knowledge about such concepts. In [3], LTN is used to annotate a reinforcement learning environment with prior knowledge and incorporate latent information into an agent. In [42], authors embed LTN in a state-of-the-art convolutional object detector. Extensions and generalizations of LTN have also been proposed in the past years, such as LYRICS [47] and Differentiable Fuzzy Logic (DFL) [68,69]. LYRICS provides an input language allowing one to define background knowledge using a first-order logic where predicate and function symbols are grounded onto any computational graph. DFL analyzes how a large collection of fuzzy logic operators behave in a differentiable learning setting. DFL also introduces new semantics for fuzzy logic implications called sigmoidal implications, and it shows that such semantics outperform other semantics in several semi-supervised machine learning tasks.

This paper provides a thorough description of the full formalism and several extensions of LTN. We show using an extensive set of explanatory examples, how LTN can be applied to solve many ML tasks with the help of logical knowledge. In particular, the earlier versions of LTN have been extended with: (1) Explicit domain declaration: constants, variables, functions and predicates are now domain typed (e.g. the constants John and Paris can be from the domain of person and city, respectively). The definition of structured domains is also possible (e.g. the domain couple can be defined as the Cartesian product of two domains of persons); (2) Guarded quantifiers: guarded universal and existential quantifiers now allow the user to limit the quantification to the elements that satisfy some Boolean condition,e.g. $\mathrm{\forall }x:\mathrm{age}\left(x\right)<10$ (playsPiano $\left(x\right)\to$ enfantProdige $\left(x\right)$ ) restricts the quantification to the cases where age is lower than 10; (3) Diagonal quantification: Diagonal quantification allows the user to write statements about specific tuples extracted in order from $n$ variables. For example,if the variables capital and country both have $k$ instances such that the $i$ -th instance of capital corresponds to the $i$ -th instance of country,one can write $\mathrm{\forall }$ Diag(capital,country) capitalOf(capital,country).

Inspired by the work of [69], this paper also extends the product t-norm configuration of LTN with the generalized mean aggregator, and it introduces solutions to the vanishing or exploding gradient problems. Finally, the paper formally defines a semantic approach to refutation-based reasoning in Real Logic to verify if a statement is a logical consequence of a knowledge base. Example 4.8 proves that this new approach can better capture logical consequences compared to simply querying unknown formulas after learning (as done in [6]).

The new version of LTN has been implemented in TensorFlow 2 [1]. Both the LTN library and the code for the examples used in this paper are available at https://github.com/ logictensornetworks/logictensornetworks

The remainder of the paper is organized as follows: In Section 2, we define and illustrate Real Logic as a fully-differentiable first-order logic. In Section 3, we specify learning and reasoning in Real Logic and its modeling into deep networks with Logic Tensor Networks (LTN). In Section 4 we illustrate the reach of LTN by investigating a range of learning problems from clustering to embedding learning. In Section 5, we place LTN in the context of the latest related work in neurosymbolic AI. In Section 6 we conclude and discuss directions for future work. The Appendix contains information about the implementation of LTN in TensorFlow 2, experimental set-ups, the different options for the differentiable logic operators, and a study of their relationship with gradient computations.

Real Logic forms the basis of Logic Tensor Networks. Real Logic is defined on a first-order language $\mathcal{L}$ with a signature that contains a set $\mathcal{C}$ of constant symbols (objects),a set $\mathcal{F}$ of functional symbols,a set $\mathcal{P}$ of relational symbols (predicates),and a set $\mathcal{X}$ of variable symbols. $\mathcal{L}$ -formulas allow us to specify relational knowledge with variables,e.g. the atomic formula is_friend $\left(v_1,v_2\right)$ may state that the person $v_1$ is a friend of the person $v_2$ ,the formula $\mathrm{\forall }x\mathrm{\forall }y\left(\text{is\_friend}\left(x,y\right)\to \text{is\_friend}\left(y,x\right)\right)$ states that the relation is_friend is symmetric,and the formula $\mathrm{\forall }x\left(\mathrm{\exists }y\left(\mathrm{Italian}\left(x\right)\wedge \text{is\_friend}\left(x,y\right)\right)\right)$ states that every person has a friend that is Italian. Since we are interested in learning and reasoning in real-world scenarios where degrees of truth are often fuzzy and exceptions are present, formulas can be partially true, and therefore we adopt fuzzy semantics.

Objects can be of different types. Similarly, functions and predicates are typed. Therefore, we assume there exists a non-empty set of symbols $\mathcal{D}$ called domain symbols. To assign types to the elements of $\mathcal{L}$ we introduce the functions $\mathbf{D},{\mathbf{D}}_{\text{in }}$ and ${\mathbf{D}}_{\text{out }}$ such that:

function $f$ or a predicate $p$ . If $f$ takes two arguments (for example, $f\left(x,y\right)$ ), ${\mathbf{D}}_{\mathbf{in}}\left(f\right)$ returns two domains, one per argument.

Real Logic may also contain propositional variables,as follows: if $P$ is a 0-ary predicate with ${\mathbf{D}}_{\mathbf{in}}\left(P\right)=⟨⟩$ (the empty sequence of domains) then $P$ is a propositional variable (an atom with truth-value in the interval $\left[0,1\right]$ ).

A term is constructed recursively in the usual way from constant symbols, variables, and function symbols. An expression formed by applying a predicate symbol to an appropriate number of terms with appropriate domains is called an atomic formula, which evaluates to true or false in classical logic and a number in $\left[0,1\right]$ in the case of Real Logic. We define the set of terms of the language as follows:

We allow the following set of formula in $\mathcal{L}$ :

We use $\diamond \in \{\mathrm{\neg }\}$ (negation), $\circ \in \{\wedge ,\vee ,\to ,\leftrightarrow \}$ (conjunction,disjunction,implication and biconditional,respectively) and $Q\in \{\mathrm{\forall },\mathrm{\exists }\}$ (universal and existential,respectively).

Example 1. Let Town denote the domain of towns in the world and People denote the domain of living people. Suppose that $\mathcal{L}$ contains the constant symbols Alice,Bob and Charlie of domain People,and Rome and Seoul of domain Town. Let $x$ be a variable of domain People and $u$ be a variable of domain Town. The term $x,u$ (i.e. the sequence $x$ followed by $u$ ) has domain People,Town which denotes the Cartesian product between People and Town (People $\times$ Town). Alice,Rome is interpreted as an element of the domain People, Town. Let lives in be a predicate with input domain ${\mathbf{D}}_{\text{in }}$ (lives_in) = People,Town. lives_in(Alice,Rome) is a well-formed expression,whereas lives_in(Bob, Charlie) is not.

The semantics of Real Logic departs from the standard abstract semantics of First-order Logic (FOL). In Real Logic, domains are interpreted concretely by tensors in the real field T Every object denoted by constants, variables, and terms, is interpreted as a tensor of real values. Functions are interpreted as real functions or tensor operations. Predicates are interpreted as functions or tensor operations projecting onto a value in the interval $\left[0,1\right]$ .

${}^1$ In the rest of the paper,we commonly use "tensor" to designate "tensor in the real field".

To emphasize the fact that in Real Logic symbols are grounded onto real-valued features, we use the term grounding,denoted by $\mathcal{G}$ ,in place of interpretation ${}^2$ . Notice that this is different from the common use of the term grounding in logic, which indicates the operation of replacing the variables of a term or formula with constants or terms containing no variables. To avoid confusion, we use the synonym instantiation for this purpose. $\mathcal{G}$ associates a tensor of real numbers to any term of $\mathcal{L}$ ,and a real number in the interval $\left[0,1\right]$ to any formula $\varphi$ of $\mathcal{L}$ . Intuitively, $\mathcal{G}\left(t\right)$ are the numeric features of the objects denoted by $t$ ,and $\mathcal{G}\left(\varphi \right)$ represents the system’s degree of confidence in the truth of $\varphi$ ; the higher the value,the higher the confidence.

A grounding for a logical language $\mathcal{L}$ on the set of domains $\mathcal{D}$ provides the interpretation of both the domain symbols in $\mathcal{D}$ and the non-logical symbols in $\mathcal{L}$ .

Definition 1. A grounding $\mathcal{G}$ associates to each domain $D\in \mathcal{D}$ a set $\mathcal{G}\left(D\right)\subseteq \bigcup _{n_1\dots n_d\in {\mathbb{N}}^{\ast }}{\mathbb{R}}^{n_1\times \cdots \times n_d}$ .

For every $D_1\dots D_n\in {\mathcal{D}}^{\ast },\mathcal{G}\left(D_1\dots D_n\right)=\underset{i=1}{\overset{n}{\times }}\mathcal{G}\left(D_i\right)$ ,that is $\mathcal{G}\left(D_1\right)\times \mathcal{G}\left(D_2\right)\times \dots \times \mathcal{G}\left(D_n\right)$ .

Notice that the elements in $\mathcal{G}\left(D\right)$ may be tensors of any rank $d$ and any dimensions $n_1\times \cdots \times n_d$ , as ${\mathbb{N}}^{\ast }$ denotes the Kleene star of $\mathbb{N}$ 3

Example 2. Let digit_images denote a domain of images of handwritten digits. If we use images of $256\times 256$ RGB pixels,then $\mathcal{G}$ (digit_images) $\subseteq {\mathbb{R}}^{256\times 256\times 3}$ . Let us consider the predicate is_digit $\left(\mathbf{Z},8\right)$ . The terms $\mathbf{Z},8$ have domains digit_images,digits. Any input to the predicate is a tuple in $\mathcal{G}$ (digit_images,digits) $=\mathcal{G}$ (digit_images) $\times \mathcal{G}$ (digits).

A grounding assigns to each constant symbol $c$ ,a tensor $\mathcal{G}\left(c\right)$ in the domain $\mathcal{G}\left(\mathbf{D}\left(c\right)\right)$ ; It assigns to a variable $x$ a finite sequence of tensors $d_1\dots d_k$ ,each in $\mathcal{G}\left(\mathbf{D}\left(x\right)\right)$ . These tensors represent the instances of $x$ . Differently from in FOL where a variable is assigned to a single value of the domain of interpretations at a time, in Real Logic a variable is assigned to a sequence of values in its domain,the $k$ examples of $x$ . A grounding assigns to a function symbol $f$ a function taking tensors from $\mathcal{G}\left({\mathbf{D}}_{\mathbf{in}}\left(f\right)\right)$ as input,and producing a tensor in $\mathcal{G}\left({\mathbf{D}}_{\mathbf{out}}\left(f\right)\right)$ as output. Finally,a grounding assigns to a predicate symbol $p$ a function taking tensors from $\mathcal{G}\left({\mathbf{D}}_{\text{in }}\left(p\right)\right)$ as input,and producing a truth-value in the interval $\left[0,1\right]$ as output.

Definition 2. A grounding $\mathcal{G}$ of $\mathcal{L}$ is a function defined on the signature of $\mathcal{L}$ that satisfies the following conditions:

${}^2$ An interpretation is an assignment of truth-values $true$ or $false$ , or in the case of Real Logic a value in [0,1], to a formula. A model is an interpretation that maps a formula to true

${}^3$ A tensor of rank 0 corresponds to a scalar,a tensor of rank 1 to a vector,a tensor of rank 2 to a matrix and so forth,in the usual way.

If a grounding depends on a set of parameters $\theta$ , we denote it as ${\mathcal{G}}_{\theta }(\cdot )$ or $\mathcal{G}\left(\cdot \mid \theta \right)$ interchangeably. Section 4 describes how such parameters can be learned using the concept of satisfiability.

We now extend the definition of grounding to all first-order terms and atomic formulas. Before formally defining these groundings, we describe on a high level what happens when grounding terms that contain free variables.

Let $x$ be a variable that denotes people. As explained in Definition 2, $x$ is grounded as an explicit sequence of $k$ instances $\left(k=\left|\mathcal{G}\left(x\right)\right|\right)$ . Consequently,a term height $\left(x\right)$ is also grounded in $k$ height values, each corresponding to one instance. We can generalize to expressions with multiple free variables, as shown in Example 3.

In the formal definition below, instead of considering a single term at a time, it is convenient to consider sequences of terms $\mathbf{t}=t_1{t}_2\dots t_k$ and define the grounding on $\mathbf{t}$ (with the definition of the grounding of a single term being derived as a special case). The fact that the sequence of terms $\mathbf{t}$ contains $n$ distinct variables $x_1,\dots ,x_n$ is denoted by $\mathbf{t}\left(x_1,\dots ,x_n\right)$ . The grounding of $\mathbf{t}\left(x_1,\dots ,x_n\right)$ ,denoted by $\mathcal{G}\left(\mathbf{t}\left(x_1,\dots ,x_n\right)\right)$ ,is a tensor with $n$ corresponding axes,one for each free variable, defined as follows:

Definition 3. Let $\mathbf{t}\left(x_1,\dots ,x_n\right)$ be a sequence $t_1\dots t_m$ of $m$ terms containing $n$ distinct variables $x_1,\dots ,x_n$ . Let each term $t_i$ in $\mathbf{t}$ contain $n_i$ variables $x_{j_{i1}},\dots ,x_{j_{i{n}_i}}$ .

If term $t_i$ contains $n_i$ variables $x_{j_1},\dots ,x_{j_{n_i}}$ selected from $x_1,\dots ,x_n$ then $\mathcal{G}{\left(t_i\right)}_{k_{j_1}\dots k_{j_{n_i}}}$ can be obtained from $\mathcal{G}{\left(\mathbf{t}\right)}_{i_1\dots i_n}$ with an appropriate mapping of indices $i$ to $k$ .

${}^4$ We assume the usual syntactic definition of free and bound variables in FOL. A variable is free if it is not bound by a quantifier $\left(\mathrm{\forall },\mathrm{\exists }\right)$ .

Figure 1: Illustration of Example 3 - $x$ and $y$ indicate dimensions associated with the free variables $x$ and $y$ . A tensor representing a term that includes a free variable $x$ will have an axis $\overrightarrow{x}$ . One can index $\overrightarrow{x}$ to obtain results calculated using each of the $v_1,v_2$ or $v_3$ values of $x$ . In our graphical convention,the depth of the boxes indicates that the tensor can have feature dimensions (refer to the end of Example 3).

Example 3. Suppose that $\mathcal{L}$ contains the variables $x$ and $y$ ,the function $f$ ,the predicate $p$ and the set of domains $\mathcal{D}=\{V,W\}$ . Let $\mathbf{D}\left(x\right)=V,\mathbf{D}\left(y\right)=W,{\mathbf{D}}_{\mathbf{in}}\left(f\right)=VW,{\mathbf{D}}_{\mathbf{out}}\left(f\right)=W$ and $\mathbf{D}\left(p\right)=VW$ . In what follows,an example of the grounding of $\mathcal{L}$ and $\mathcal{D}$ is shown on the left,and the grounding of some examples of possible terms and atomic formulas is shown on the right.

Notice the dimensions of the results. $\mathcal{G}\left(f\left(x,y\right)\right)$ and $\mathcal{G}\left(p\left(x,f\left(x,y\right)\right)\right)$ return $\left|\mathcal{G}\left(x\right)\right|\times \left|\mathcal{G}\left(y\right)\right|=3\times 2$ values, one for each combination of individuals that occur in the variables. For functions, we can have additional dimensions associated to the output domain. Let us suppose a different grounding such that $\mathcal{G}\left({\mathbf{D}}_{\text{out }}\left(f\right)\right)={\mathbb{R}}^m$ . Then the dimensions of $\mathcal{G}\left(f\left(x,y\right)\right)$ would have been $\left|\mathcal{G}\left(x\right)\right|\times \left|\mathcal{G}\left(y\right)\right|\times m$ ,where $\left|\mathcal{G}\left(x\right)\right|\times \left|\mathcal{G}\left(y\right)\right|$ are the dimensions for indexing the free variables and $m$ are dimensions associated to the output domain of $f$ . Let us call the latter feature dimensions, as captioned in Figure 1. Notice that $\mathcal{G}\left(p\left(x,f\left(x,y\right)\right)\right)$ will always return a tensor with the exact dimensions $\left|\mathcal{G}\left(x\right)\right|\times \left|\mathcal{G}\left(y\right)\right|\times 1$ because,under any grounding,a predicate always returns a value in $\left[0,1\right]$ . Therefore,as the "feature dimensions" of predicates is always 1,we choose to "squeeze it" and not to represent it in our graphical convention (see Figure 1, the box output by the predicate has no depth).

Figure 2: Illustration of an element-wise operator implementing conjunction $\left(p\left(x\right)\wedge q\left(y\right)\right)$ . We assume that $x$ and $y$ are two different variables. The result has one number in the interval $\left[0,1\right]$ to every combination of individuals from $\mathcal{G}\left(x\right)$ and $\mathcal{G}\left(y\right)$ .

The semantics of the connectives is defined according to the semantics of first-order fuzzy logic [28]. Conjunction $(\wedge )$ ,disjunction $(\vee )$ ,implication $(\to )$ and negation $(-)$ are associated, respectively,with a t-norm $\left(T\right)$ ,a t-conorm $\left(S\right)$ ,a fuzzy implication $\left(I\right)$ and a fuzzy negation $\left(N\right)$ operation FuzzyOp $\in \{T,S,I,N\}$ . Definitions of some common fuzzy operators are presented in Appendix B Let $\varphi$ and $\psi$ be two formulas with free variables $x_1,\dots ,x_m$ and $y_1,\dots ,y_n$ ,respectively. Let us assume that the first $k$ variables are common to $\varphi$ and $\psi$ . Recall that $\diamond$ and $\circ$ denote the set of unary and binary connectives, respectively. Formally:

In (2), $\left(i_1,\dots ,i_k\right)$ denote the indices of the $k$ common variables, $\left(i_{k+1},\dots ,i_m\right)$ denote the indices of the $m-k$ variables appearing only in $\varphi$ ,and $\left(i_{m+1},\dots ,i_{m+n-k}\right)$ denote the indices of the $n-k$ variables appearing only in $\psi$ . Intuitively, $\mathcal{G}\left(\varphi \circ \psi \right)$ is a tensor whose elements are obtained by applying $\mathrm{FuzzyOp}(\circ )$ element-wise to every combination of individuals from $x_1,\dots ,x_m$ and $y_1,\dots ,y_n$ (see Figure 2).

The semantics of the quantifiers $\left(\{\mathrm{\forall },\mathrm{\exists }\}\right)$ is defined with the use of aggregation. Let Agg be a symmetric and continuous aggregation operator, $\mathrm{Agg}:\bigcup _{\prec N}{[0,1]}^n\to \left[0,1\right]$ . An analysis of suitable